Home | Project Team | Project Description | State Results |Regional Results| Executive Summary

Home | Project Team | Project Description | State Results |Regional Results| Executive Summary

Sustaining Evidence-Based Practices –
What We Are Learning about the Michigan Experience
Executive Summary
October 2010

The major objective of the evaluation study conducted during fiscal year 2009-2010 was to explore organizational factors that have influenced the sustainability of the evidence-based practices (Integrated Dual Disorders Treatment (IDDT), Family Psychoeducation (FPE) and Parent Management Training-Oregon (PMTO)) and the integration of these practices and other evidence-based practices (EBPs) throughout the state.  

The three evidence-based practices (IDDT, FPE and PMTO) were the initial practices identified by the State for targeted grants to help regions to implement one of these EBPs. The initial state grants provided key funds to support training, supervision and certification of staff in the evidence-based practices.  The grants were awarded to regions for two years covering Fiscal Year 2005-2006 and 2006-2007.  The evaluation study involved reviewing the initial state grant awards and quarterly reports submitted from the regions during the two year grant cycle.  The review of these proposals and quarterly reports was to determine whether the regions were successful in implementing the EBPs selected and to note any challenges that the regions identified in implementing the EBPs with fidelity.

Focus Groups and Key Informant Interviews:   Next, the evaluation team conducted focus groups with the state-mandated Improving Practices Leadership Teams (IPLT) at the regional level.  A total of 16 of the 18 regions participated in the focus group sessions.   The IPLT teams have representatives from the key EBPs, administrators, supervisors, consumers, and finance/budgeting administrators. The purpose of the focus groups was to examine implementation and sustainability of the evidence based practices, in particular IDDT, FPE and PMTO. We also engaged the IPLT members in exploring how each team functioned and the current role of the IPLT in the implementation and sustainability of EBPs.  The data from the focus groups was transcribed and analyzed for key themes and issues.  These key themes and issues were used to develop the statewide survey.

In addition, the evaluation team also conducted 49 key informant interviews with representatives identified by the regions as individuals who were involved in the implementation and sustainability efforts of the EBPs.  These key informant interviews were transcribed and analyzed for key themes and issues.  The findings from the key informant interviews were also incorporated into the statewide survey questions.
The major themes that emerged in the review of the 16 IPLT focus groups and the 49 key informant interviews included:

State Level:

• At the State Level, the identification of a key point person for each EBP supported by the state is viewed as an important link for the region.  This person helps to address the implementation and sustainability issues at a statewide level and to assist in the coordination of efforts statewide related to the EBP.
• The State established a statewide Improving Practices Leadership Team with representatives from each region.  Greater clarification about the role of this team and State as it relates to sustaining the EBPs was seen by the respondents as important, especially in advocating for changes needed to support EBPs over time.  Some of the advocacy areas included: 
• increase awareness of the need to pace the implementation of EBPs to better support regions;
•  address the reality that EBPs are “add ons” to workload for many staff who volunteer to be trained and deliver the EBPs;
• examine ways to adapt EBPs to meet regional needs, especially when one considers rural and urban differences, and regions that operate as a single county versus multi county sites;
• explore different supervision and EBP certification processes to maximize the number of staff and supervisors that can deliver the EBPs and keep up with staff turnover;
• propose ways to support transportation issues for consumers and staff with EBPs that require extensive outreach; and
• promote more graduate programs in social work/psychology/allied health fields to teach EBPs so when students graduate from programs they are ready to engage in the next steps for EBP.
• The state is viewed by the regions as the funding source to support start up of new EBPs and the implementer of the certification requirements and billing procedures for each EBP.  This has been a challenge at times for regions since funding through the grant mechanisms is limited, new billing codes take time to put into place, and certifying who can bill is not always clear. 
• The state structures are also viewed by respondents as a barrier in the implementation of new EBPs due to the strict fidelity requirements that need to be met in order to implement an EBP and the limited number of EBPs approved by the state for initial funding.  Many regions would like to see greater flexibility in the EBPs selected and would like more local choice about what EBPs to implement.  The State helping to establish guidelines for adapting EBPs to local needs was an area that was mentioned often by respondents.
• In terms of sustaining EBPs, the regions look to the state for funding to continue to train workers in the EBPs and initial supervision in the EBPs for workers to be certified.  Delivering the trainings at local sites is a high priority for many regions. The high staff turnover and need for frequent trainings was an area that the respondents look to the State for assistance.  It is a challenge for the regions to keep workers who have been trained and certified to deliver the EBPs.  This is especially a problem for some of the EBPs that require extensive supervision prior to certification.  The train-the-trainer model has worked in regions where there are multiple staff trained in an EBPs but for several regions this approach is costly due to staff size and time away from regular duties for staff members who are trained to be trainers.
• The state audit/quality improvement reporting activities often duplicate work currently being done at the regional level related to fidelity of the EBPs.  Development of common outcome measures at the state level for EBPs is another area that would enhance the ability for regions to determine what EBPs should be sustained as new EBPs emerge. Many respondents recommended that the State engage regions in a discussion about the development of a statewide electronic medical record system.

Regional Level (PIHP):

From the focus groups and key information interviews, the data suggest that the IPLT teams serve the regions as the key organizational structure for the implementation and sustainability of the EBPs in the region. Most IPLT teams reported that they are responsible for monitoring all EBPs in the region, which may total up to 7 or more EBPs.

• The IPLT teams are viewed by the regions as critical to helping to sustain the EBPs.   Some regions have modified the IPLT membership to reflect the local needs of the region (e.g. adding key supervisors from all EBPs, representatives from each mental health center, contract agency representatives).  The IPLT teams in most regions take primary responsibility for
• oversight of all EBPs operating in the region,
• strategic planning activities related to adding additional EBPs, and
• monitoring fidelity to EBPs currently operating in the region. 
• Most of the IPLT teams make recommendations related to EBP implementation and sustainability directly to the administration or leadership of the region.  
• The IPLT teams are operating within each region as the central team to address EBP issues.   The IPLT members feel that the work they do is critical to sustaining the EBPs and that while some of the same people are also on other key organizational committees; this is the only committee where the entire focus is on EBPs.   In several regions, the IPLT teams also provide the ongoing communication about EBPs to all internal and contract agencies in the region.    
• In terms of how meetings are organized, most of the IPLTs include a reporting component to update all members on the status of each EBP and a problem-solving component to address challenges faced in keeping fidelity or implementing the EBPs.   Several of the IPLT teams have sub-committees for each EBP that works on the day to day challenges of implementing/ sustaining the EBPs.  The IPLT teams in many regions are also beginning to focus more on gathering data to assess consumer outcomes related to the implementation of the EBPs. 
• For every IPLT team, one of the challenges has been to try to provide oversight on all of the EBPs that have been implemented since 2005 in the regions.  The IPLT teams highlighted the tension that has emerged for them between trying to balance the reporting out function of the committee with the problem-solving and ongoing sustainability functions of the committee.  Each region has worked to address time constraints on team members and the frequency of IPLT meetings.  The concern is that as EBPs continue to be implemented, the IPLT team may not be able to meet all the responsibilities for oversight of EBPs.
• Several of the IPLT teams mentioned the need to recognize through the reward structure in the organization, staff members who become “champions” for the EBPs.  These staff members often take on additional workload tasks in order to support the EBP growth in the regions and in the state.

 

Statewide Survey:  The statewide survey on organizational factors related to sustainability of the EBPs was developed from the key informant and focus group information.  The statewide survey asked IPLT participants and key informants to rank items that emerged from the IPLT focus groups and key informant interviews related to organizational factors for sustainability and also included survey items that addressed additional areas critical to the sustainability of EBPs.  The ranking items involved the following areas:  choosing the EBPs; potential statewide solutions to challenges in sustaining EBPs; fidelity; training; staffing; and gathering and measuring outcomes.  Additional questions that required a 7 point Likert response (strongly agree to strongly disagree) were also asked.  The additional questions included more information on fidelity factors, training factors, gathering and measuring outcomes, geographic location, technology and awareness of EBPs. 

A total of 190 individuals received this statewide survey and a total of 139 individuals completed the survey.  Sixteen of Michigan’s 18 PIHPs were represented, with response rates ranging from 50 percent to 88.8 percent for each PIHP.  

The key findings from this statewide survey as it relates to factors related to sustaining EBPs include:

• The top critical factors for PIHPs choosing and sustaining EBPs included consumer demand (53.2%), availability of ongoing funding from the state (15.1%), and ability to maintain fidelity to the model (8.6%).
• When ranking items related to statewide solutions to the challenges in sustaining EBPs, the respondents ranked highest the need to fund regions to select EBPs based on local needs (46.4%).   This was followed by increasing consumer representation in the selection of EBPs (13%).  The third solution that was supported by the respondents focused on the certification process with respondents indicating that it is important to certify the CMHPS/Region not just individual staff for EBPs (12.3%).
• Promoting adaptations of the EBPs to accommodate regional differences was rated most important to respondents in terms of maintaining fidelity to the EBPs (46.3%).  Respondents also rated establishment of more flexible certification standards as the second most important solution to helping regions sustain the EBPs (35.1%).
• When considering staffing needs for sustaining EBPs, the respondents rated three items as the most important:  ensuring that EBPs do not add to staff workload (26.5%), limiting the number of EBPs a CMHSP/Region is implementing at any given time (24.3%), and developing mechanisms to certify supervisors in multiple EBPs (22.1%).
• Establishing common outcome measures for EBPs was identified by respondents as the top-ranked statewide solution concerning gathering data and measuring outcomes (48.9%).  The second most important solution for this area was to use cost-benefit analysis to determine the effectiveness of multiple EBPs (32.6%).
• The respondents noted the following challenges in sustaining EBPs: 
• developing an effective outreach policy at the PIHP level
• finding consumers who meet criteria to participate in the EBPs
• securing clinician/staff certification in order to bill for the EBPs
• keeping up with the changes in billing procedures related to EBPs
• not being able to adapt the EBPs based on regional needs
• difficulty in monitoring external provider contract agencies for EBP fidelity
• holding trainings outside the region is a significant barrier
• need for more frequent, ongoing EBP trainings
• addressing cultural and diversity factors in the EBP trainings
• difficulty in documenting how EBPs benefit consumers
• need to reduce duplication of reporting EBP fidelity (maybe incorporate into audits and QI process)
• lack of standardized outcome measures for EBPs
• transportation issues for consumers and staff limit ability to deliver EBPs
• recruiting staff with the required State certification for specific EBPs
• funding for networking and communication technology
• need to increase onsite training opportunities (e.g. DVDs) and conferencing technologies
• When looking at differences in responses using the t-test for difference between predominately rural regions to predominately urban regions, the results suggest that for most items there are not significant differences.  The following areas did show significant differences based on the rural/urban categories:
• Predominately rural regions would put greater importance than predominately urban regions on choosing EBPs based on the easiness of the certification process.
• Predominately rural regions would put greater weight than predominately urban regions on selected EBPs that do not increase staff workload.
• Predominately rural regions placed greater importance on establishing common outcome measures for EBPs when compared to the predominately urban regions.
• One-way ANOVA analyses were used to test for differences in the responses based on number of counties served by a region (single county; multi-county (2-6); multi-county (7-15).  There were significant differences found across the three groups in only two areas:  staff workload issues and certification of supervisors in multiple EBPs.  The larger regions with more counties indicated a greater need to ensure that staff workload was not increased due to the EBP implementation. The single county region indicated a greater need to certify supervisors in multiple EBPs.

 

 

 

For further information about the evaluation study, please contact:

Mary C. Ruffolo, Ph.D., LMSW
University of Michigan School of Social Work
1080 South University
Ann Arbor, MI 48109
(734)936-4799
mruffolo@umich.edu

 

The study was funded though a Medicaid Match grant mechanism and was awarded to the University of Michigan Department of Psychiatry and the School of Social Work.  Dave Neal, LMSW and Karen Milner, M.D. are the principal investigators of the overall grant mechanism.  Mary C. Ruffolo, Ph.D. and Brian Perron, Ph.D. were the primary investigators of this evaluation study on sustainability of evidence-based practices. Stephanie Benson, MSW was the project coordinator. The University of Michigan School of Social Work Curtis Center also funded portions of this study.

Alyson Rush from the Department of Community Mental Health served as the state liaison to this study.